{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在第三节中，计算梯度使用的是求损失函数关于参数的导数。优点是实现起来比较方便，但存在致命问题：计算太耗时。为了改进这种求梯度的方法，本节采用**误差反向传播法**。\n",
    "\n",
    "要正确理解误差反向传播法，我个人认为有两种方法：一种是基于**数学式**；另一种是基于**计算图（computational graph）**。前者是比较常见的方法，机器学习相关的图书中多数都是以数学式为中心展开论述的。因为这种方法严密且简洁，所以确实非常合理，但如果一上来就围绕数学式进行探讨，会忽略一些根本的东西，止步于式子的罗列。因此，本章希望大家通过计算图，直观地理解误差反向传播法。\n",
    "\n",
    "# 一、计算图\n",
    "\n",
    "计算图：表示计算过程的图。为了熟悉计算图，先用几个简单的例子体会一下计算图的作用。\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1计算图初步理解\n",
    "\n",
    "计算图通过箭头和节点表示计算过程，**节点中存放计算的内容，箭头上方存放中间结果**。\n",
    "\n",
    ">【例题1】小明在商店买了2个单价为100元的苹果，税率是10％，计算总的支付金额。\n",
    "\n",
    "建立的计算图如下：\n",
    "\n",
    "![image-20230323201816517](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230323201816517.png)\n",
    "\n",
    "或者\n",
    "\n",
    "![image-20230323201921890](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230323201921890.png)\n",
    "\n",
    ">【例题2】小明在商店买了2个苹果、3个橘子。苹果单价100元，橘子单价150元。消费税10％，计算总支付金额。\n",
    "\n",
    "![image-20230323202225964](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230323202225964.png)\n",
    "\n",
    "综上，利用计算图解决问题的步骤如下：\n",
    "\n",
    "1. 构建计算图\n",
    "\n",
    "2. 对计算图从左向右进行计算。\n",
    "\n",
    "上述的“从左向右”也称为**正向传播**，即从出发点到结束点的传播；从右向左也成为**反向传播**，反向传播对后面的求导数有至关重要的作用。\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2局部计算\n",
    "\n",
    "**局部计算**是指，无论全局发生了什么，都能只根据与自己相关的信息输出接下来的结果。\n",
    "\n",
    "对局部计算的理解以下面这张图为例：\n",
    "\n",
    "![image-20230323202935280](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230323202935280.png)\n",
    "\n",
    "图中，购买了很多物品，苹果买了两个，其他物品总价4000。这里的重点是，各个节点处的计算都是局部计算。这意味着，例如苹果和其他很多东西的求和运算（4000 + 200 → 4200）并不关心4000这个数字是如何计算而来的，只要把两个数字相加就可以了。换言之，各个节点处只需进行与自己有关的计算（在这个例子中是对输入的两个数字进行加法运算），不用考虑全局。\n",
    "\n",
    "因此，无论全局的计算有多么复杂，各个步骤所要做的就是对象节点的局部计算。虽然局部计算非常简单，但是通过传递它的计算结果，可以获得全局的复杂计算的结果。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3利用计算图的反向传播解决导数计算\n",
    "\n",
    "计算图的三个优点：\n",
    "\n",
    "1. 全局 → 局部，复杂 → 简单\n",
    "\n",
    "2. 保存中间结果\n",
    "\n",
    "3. 反向传播计算导数\n",
    "\n",
    "还是以例题一为例：\n",
    "\n",
    ">【例题1】小明在商店买了2个单价为100元的苹果，税率是10％，计算总的支付金额。\n",
    "\n",
    "问：单价的变化会对总价造成多少影响？假设总价为$L$，单价为$x$。\n",
    "\n",
    "解：总价对单价求导数，即$\\frac{\\partial L}{\\partial x}$，该数值可以通过计算图的反向传播来计算。\n",
    "\n",
    "![image-20230323220409279](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230323220409279.png)\n",
    "\n",
    "如上图所示，反向传播传递**局部导数**，将导数的值写在箭头下方（1 → 1.1 → 2.2）。从结果中可以看出，“支付金额关于单价的导数”值为2.2，意思是：如果单价上涨1元，总价就会涨2.2元。同样的，关于“税率”的导数也可以用这个方法求出来。并且，**中途计算出来的导数结果可以共享**，从而高效计算多个导数。"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# 二、链式法则\n",
    "\n",
    "反向传播局部导数的原理，是基于**链式法则**。\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1计算图的反向传播\n",
    "\n",
    "![image-20230323223026675](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230323223026675.png)\n",
    "\n",
    "反向传播的计算顺序：$信号E×节点的局部导数$，传递到下一个节点。\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2链式法则\n",
    "\n",
    "链式法则就是一个关于复合函数求导的性质：\n",
    "\n",
    ">如果某个函数由复合函数表示，则该复合函数的导数可以用构成复合函数的各个函数的导数的乘积表示。\n",
    "\n",
    "例如，现有两个函数：\n",
    "\n",
    "$$\n",
    "    z=t^2\\\\\n",
    "\n",
    "    t=x+y\n",
    "$$\n",
    "\n",
    "要求$z$关于$x$的导数，即$\\frac{\\partial z}{\\partial x}$，就要先求$z$关于$t$的导数，再求$t$关于$x$的导数，最后把二者相乘，即：\n",
    "\n",
    "$$\\begin{aligned}\n",
    "    \\frac{\\partial z}{\\partial x}\n",
    "    &=\\frac{\\partial z}{\\partial t}\\frac{\\partial t}{\\partial x}\\\\\n",
    "    &=2t*1\\\\\n",
    "    &=2(x+y)\n",
    "\\end{aligned}$$\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3链式法则与计算图\n",
    "\n",
    "将上述的计算过程用计算图表示出来：\n",
    "\n",
    "![image-20230324092812052](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230324092812052.png)\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.4反向传播\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4.1加法节点的反向传播\n",
    "\n",
    "以$z=x+y$为例，计算图的反向传播如下，假设$\\frac{\\partial L}{\\partial z}$是上游传过来的导数：\n",
    "\n",
    "![image-20230324095753167](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230324095753167.png)\n",
    "\n",
    "所以加法节点的计算图可以抽象成如下的形式：\n",
    "\n",
    "![image-20230324100012979](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230324100012979.png)\n",
    "\n",
    "现在代入具体的数据来计算，假设有\"10+5=5\"这一加法计算，反向传播时，上游传过来导数值1.3，如下图：\n",
    "\n",
    "![image-20230324100325067](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230324100325067.png)\n",
    "\n",
    "所以，加法节点的反向传播是：**怎么传进来就怎么传出去**。\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4.2乘法节点的反向传播\n",
    "\n",
    "以$z=xy$为例，计算图的反向传播如下：\n",
    "\n",
    "![image-20230324103703866](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230324103703866.png)\n",
    "\n",
    "代入具体的数据看：\n",
    "\n",
    "![image-20230324103804828](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230324103804828.png)\n",
    "\n",
    "所以，乘法节点的反向传播是：**传进来的数乘另一个数再传出去**。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4.3苹果的例子再回忆\n",
    "\n",
    "![image-20230324105246732](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230324105246732.png)\n",
    "\n",
    "根据加法和乘法节点的反向传播的例子，很容易就能解决该图中的导数问题了。"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 三、简单层的实现\n",
    "\n",
    "使用python实现之前买苹果的例子。这里，我们把要实现的计算图的**乘法节点**称为“乘法层”（MulLayer），**加法节点**称为“加法层”（AddLayer）。\n",
    "\n",
    "![image-20230324123905184](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230324123905184.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1乘法层的实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "220.00000000000003\n",
      "1.1 200 2.2 110.00000000000001\n"
     ]
    }
   ],
   "source": [
    "class MulLayer:\n",
    "    # x,y保存正向传播的输入\n",
    "    def __init__(self):\n",
    "        self.x = None\n",
    "        self.y = None\n",
    "\n",
    "    def forward(self, x, y):\n",
    "        self.x = x\n",
    "        self.y = y\n",
    "        out = x*y\n",
    "\n",
    "        return out\n",
    "\n",
    "    def backward(self, dout):\n",
    "        dx = dout*self.y\n",
    "        dy = dout*self.x\n",
    "\n",
    "        return dx, dy\n",
    "\n",
    "\n",
    "# 正向传播测试\n",
    "price_per_apple = 100\n",
    "apple_num = 2\n",
    "tax = 1.1\n",
    "\n",
    "mul_apple_layer = MulLayer()\n",
    "mul_tax_layer = MulLayer()\n",
    "\n",
    "apple_price = mul_apple_layer.forward(price_per_apple, apple_num)\n",
    "total_price = mul_tax_layer.forward(apple_price, tax)\n",
    "\n",
    "print(total_price)\n",
    "\n",
    "# 反向传播测试\n",
    "dtotal_price = 1\n",
    "dapple_price, dtax = mul_tax_layer.backward(dtotal_price)\n",
    "dprice_per_apple, dapple_num = mul_apple_layer.backward(dapple_price)\n",
    "\n",
    "print(dapple_price, dtax, dprice_per_apple, dapple_num)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2加法层的实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class AddLayer:\n",
    "    def __init__(self) -> None:\n",
    "        pass\n",
    "\n",
    "    def forward(self, x, y):\n",
    "        return x+y\n",
    "\n",
    "    def backward(self, dout):\n",
    "        return dout, dout\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3加法乘法结合实现\n",
    "\n",
    "用MulLayer和AddLayer实现如下的计算图：\n",
    "\n",
    "![image-20230324140509690](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230324140509690.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "715.0000000000001\n",
      "110.00000000000001 2.2 3.3000000000000003 165.0 650\n"
     ]
    }
   ],
   "source": [
    "# 定义参数\n",
    "apple_num = 2\n",
    "orange_num = 3\n",
    "price_per_apple = 100\n",
    "price_per_orange = 150\n",
    "tax = 1.1\n",
    "\n",
    "# 定义加法和乘法层\n",
    "mul_apple_layer = MulLayer()\n",
    "mul_orange_layer = MulLayer()\n",
    "add_fruit_layer = AddLayer()\n",
    "mul_price_layer = MulLayer()\n",
    "\n",
    "# 前向传播\n",
    "apple_price = mul_apple_layer.forward(apple_num, price_per_apple)  # apple层\n",
    "orange_price = mul_orange_layer.forward(\n",
    "    orange_num, price_per_orange)  # orange层\n",
    "fruit_price = add_fruit_layer.forward(apple_price, orange_price)  # fruit层\n",
    "total_price = mul_price_layer.forward(fruit_price, tax)  # total_price层\n",
    "print(total_price)\n",
    "\n",
    "# 反向传播\n",
    "dtotal_price = 1\n",
    "dfruit_price, dtax = mul_price_layer.backward(dtotal_price)  # total_price层\n",
    "dapple_price, dorange_price = add_fruit_layer.backward(dfruit_price)  # fruit层\n",
    "dorange_num, dprice_per_orange = mul_orange_layer.backward(\n",
    "    dorange_price)  # orange层\n",
    "dapple_num, dprice_per_apple = mul_apple_layer.backward(dapple_price)  # apple层\n",
    "\n",
    "print(dapple_num, dprice_per_apple, dprice_per_orange, dorange_num, dtax)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "观察上述过程发现：反向传播和正向传播的顺序是刚好相反的。\n",
    "\n",
    "## 3.4激活函数层的实现\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4.1Relu层\n",
    "\n",
    "Relu激活函数表达式如下：\n",
    "\n",
    "![image-20230325092414397](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230325092414397.png)\n",
    "\n",
    "其导数如下（虽然在0点导数不存在）：\n",
    "\n",
    "![image-20230325092534886](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230325092534886.png)\n",
    "\n",
    "根据这两个式子，画出Relu激活函数和其导数的计算图：\n",
    "\n",
    "![image-20230325092619997](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230325092619997.png)\n",
    "\n",
    "代码实现："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Relu:\n",
    "    def __init__(self) -> None:\n",
    "        self.mask = None  # 输入x是一个ndarray（由之前苹果例子中的单输入变为多输入），mask是一个布尔值数组，对应x中大于0的元素为True，其余为False\n",
    "\n",
    "    def forward(self, x):\n",
    "        self.mask = (x <= 0)\n",
    "        x[self.mask] = 0  # <=0的地方等于0\n",
    "        out = x  # 增加可读性\n",
    "\n",
    "        return out\n",
    "\n",
    "    def backward(self, dout):\n",
    "        dout[self.mask] = 0  # backward之前必然经过一遍forward，所以self.mask已经改变\n",
    "        dx = dout\n",
    "\n",
    "        return dx\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ReLU层的作用就像电路中的开关一样。正向传播时，有电流通过的话，就将开关设为 ON；没有电流通过的话，就将开关设为 OFF。反向传播时，开关为ON的话，电流会直接通过；开关为OFF的话，则不会有电流通过。\n",
    "\n",
    "### 3.4.2Sigmoid层\n",
    "\n",
    "sigmoid函数式：\n",
    "\n",
    "![image-20230325103601453](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230325103601453.png)\n",
    "\n",
    "计算图表示如下：\n",
    "\n",
    "![image-20230325104039156](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230325104039156.png)\n",
    "\n",
    "正向传播没什么好说的，反向传播从右往左依次计算导数：\n",
    "\n",
    "**（1）'/'节点**\n",
    "\n",
    "![image-20230325105149269](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230325105149269.png)\n",
    "\n",
    "求$y=\\frac{1}{x}$的导数，$\\frac{\\partial y}{\\partial x}=-\\frac{1}{x^2}=-y^2$\n",
    "\n",
    "**（2）'+'节点**\n",
    "\n",
    "![image-20230325171311826](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230325171311826.png)\n",
    "\n",
    "加法节点直通。\n",
    "\n",
    "**（3）'exp'节点**\n",
    "\n",
    "![image-20230325171434970](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230325171434970.png)\n",
    "\n",
    "求$y=e^{x}$的导数：$\\frac{\\partial y}{\\partial x}=e^{x}$\n",
    "\n",
    "**（4）'*'节点**\n",
    "\n",
    "![image-20230325171655637](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230325171655637.png)\n",
    "\n",
    "交换相乘。\n",
    "\n",
    "最后，y关于x的导数为：\n",
    "\n",
    "![image-20230325172132735](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230325172132735.png)\n",
    "\n",
    "另外，导数还可以做进一步整理：\n",
    "\n",
    "sigmoid函数式：\n",
    "\n",
    "$$\n",
    "    y=\\frac{1}{1+e^{-x}}\n",
    "$$\n",
    "\n",
    "上述导数式可以如下改写：\n",
    "\n",
    "$$\\begin{aligned}\n",
    "    \\frac{\\partial L}{\\partial y}y^2e^{-x}\n",
    "    &=\\frac{\\partial L}{\\partial y}\\frac{1}{(1+e^{-x})^2}e^{-x}\\\\\n",
    "    &=\\frac{\\partial L}{\\partial y}\\frac{1}{1+e^{-x}}\\frac{e^{-x}}{1+e^{-x}}\\\\\n",
    "    &=\\frac{\\partial L}{\\partial y}y(1-y)\n",
    "\\end{aligned}$$\n",
    "\n",
    "代码实现："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "\n",
    "class Sigmoid:\n",
    "    def __init__(self) -> None:\n",
    "        self.out = None  # 用out保存输出结果，因为在backward中会用到\n",
    "\n",
    "    def forward(self, x):\n",
    "        self.out = 1/(1+np.exp(-x))\n",
    "\n",
    "        return self.out\n",
    "\n",
    "    def backward(self, dout):\n",
    "        dx = dout*self.out*(1.0-self.out)\n",
    "\n",
    "        return dx\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4.3Affine/Softmax层\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.4.3.1Affine层\n",
    "\n",
    "在神经网络计算中，需要计算加权和：$X·W+B$。神经网络的正向传播中进行的矩阵的乘积运算在几何学领域被称为“仿射变换”（几何中，仿射变换包括一次线性变换和一次平移，分别对应神经网络的加权和运算与加偏置运算）。\n",
    "\n",
    "所以，仿射变换也能用计算图表示出来，并称其为 **“Affine”层**。\n",
    "\n",
    "![image-20230325175857572](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230325175857572.png)\n",
    "\n",
    "其中，这里假设$X、W、B$ 分别是形状为$(2,)、(2, 3)、(3,)$的多维数组。\n",
    "看其反向传播：\n",
    "\n",
    "![image-20230325181326013](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230325181326013.png)\n",
    "\n",
    "这里不对该求导公式的做深入的理解和推导，但要记住该公式。\n",
    "\n",
    "**对于矩阵乘法`W·X = Y`,矩阵Y关于矩阵W的导数为`X^T`**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.4.3.2batch版本的Affine层\n",
    "\n",
    "N个数据一起进行正向传播，就是把$X$的第一维度的值改成$N$。\n",
    "\n",
    "![image-20230326092716142](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230326092716142.png)\n",
    "\n",
    "需要注意的是，对偏置项的求导需要对反向传播的输入$\\frac{\\partial L}{\\partial Y}$按列进行求和。\n",
    "\n",
    "Affine层的实现："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "\n",
    "class Affine:\n",
    "    def __init__(self, W, b) -> None:\n",
    "        self.W = W\n",
    "        self.b = b\n",
    "        self.X = None\n",
    "\n",
    "        # 为什么要保存dW和db?\n",
    "        # 因为反向传播的输入只有dout，输出只有dX（即数据流通是靠dX实现的），\n",
    "        # dW和db是附加产物，但在更新参数的时候会用到，所以放在类属性里面保存，而不作为backward的返回值\n",
    "        self.dW = None\n",
    "        self.db = None\n",
    "\n",
    "    def forward(self, X):\n",
    "        self.X = X\n",
    "        X_dot_W = np.dot(self.X, self.W)\n",
    "        out = X_dot_W+self.b\n",
    "\n",
    "        return out\n",
    "\n",
    "    def backward(self, dout):\n",
    "        self.db = np.sum(dout, axis=0)\n",
    "        dX = np.dot(dout, self.W.T)\n",
    "        self.dW = np.dot(self.X.T, dout)\n",
    "\n",
    "        return dX\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.4.3.3Softmax-with-Loss层\n",
    "\n",
    ">神经网络分为训练阶段（学习阶段）和测试阶段（推理阶段）。因为学习阶段需要用网络的输出精确计算损失函数的值从而求梯度更新参数，需要正规化，因此在学习阶段需要用到softmax函数。但在测试阶段，由于输出最大值即为网络的预测结果，所以对输出只要取最大的即可，没必要再求softmax比例。因此，一般情况下，**测试阶段基本不用经过softmax**。\n",
    "\n",
    "![image-20230326110413093](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230326110413093.png)\n",
    "\n",
    "上图也可以简化为：\n",
    "\n",
    "![image-20230326110531479](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230326110531479.png)\n",
    "\n",
    "这里解释一下softmax层反向传播的推导：\n",
    "\n",
    "**【步骤一】** 反向传播从上游传过来的值是$-\\frac{t_1}{y_1}$，其中$t_1$是标签的第一位，第一步先经过一个乘法节点，计算出其中一个分支为$-\\frac{t_1}{y_1}·exp(a_1)=-\\frac{t_1}{exp(a_1)·\\frac{1}{S}}·exp(a_1)=-t_1S$。\n",
    "\n",
    "![在这里插入图片描述](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/daedb02436dd4f56b6cb41c147ffca96.png)\n",
    "\n",
    "**【步骤二】** **正向传播时若有分支流出，则反向传播时，他们的传播值会相加，再经过节点**。所以，在本例中除法节点之后有三分支，所以反向传播时会相加，即$(-t_1S)+(-t_2S)+(-t_3S)=-(t_1+t_2+t_3)S$。又因为$t_1,t_2,t_3$是标签（one-hot），所以其中只有一个是1，其余都为0，即得$t_1+t_2+t_3=1$，所以$(-t_1S)+(-t_2S)+(-t_3S)=-(t_1+t_2+t_3)S=-S$。然后，经过除法节点，有$-S·-(\\frac{1}{S^2})=\\frac{1}{S}$。\n",
    "\n",
    "![在这里插入图片描述](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/b7ff1fe77dba4c609ce42ba2c5143b5c.png)\n",
    "\n",
    "**【步骤三】** 遇到加法节点直通。\n",
    "\n",
    "![img](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/13fa5234fd2a4615aa1f9b8b402596fc.png)\n",
    "\n",
    "**【步骤四】** 之前乘法节点的另一分支，$-\\frac{t_1}{y_1}·\\frac{1}{S}=-\\frac{t_1}{exp(a_1)·\\frac{1}{S}}·\\frac{1}{S}=-\\frac{t_1}{exp(a_1)}$。\n",
    "\n",
    "![img](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/9911533508634c968bb7f3fa15aa84fa.png)\n",
    "\n",
    "**【步骤五】** $exp$节点两分支分流，所以反向传播时两分支的传播值要相加，即$-\\frac{t_1}{exp(a_1)}+\\frac{1}{S}$，再经过$exp$节点，$(-\\frac{t_1}{exp(a_1)}+\\frac{1}{S})·exp(a_1)=-t_1+y_1$\n",
    "\n",
    "![在这里插入图片描述](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/0e0d2205759c4e6ba4c515d28310a214.png)\n",
    "\n",
    "综上，softmax层的反向传播得到结果 **$(y_1-t_1,y_2-t_2,y_3-t_3)$**，这个结果非常漂亮。\n",
    ">实际上，这样“漂亮”的结果并**不是偶然的**，而是为了得到这样的结果，**特意设计了交叉熵误差函数**。回归问题中输出层使用“恒等函数”，损失函数使用“平方和误差”，也是出于同样的理由。也就是说，使用“平方和误差”作为“恒等函数”的损失函数，反向传播才能得到（y1 −t1, y2 − t2, y3 − t3）这样“漂亮”的结果。\n",
    "\n",
    "下面来实现Softmax-with-Loss层："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def softmax(x):\n",
    "    if x.ndim == 2:\n",
    "        x = x.T\n",
    "        x = x - np.max(x, axis=0)\n",
    "        y = np.exp(x) / np.sum(np.exp(x), axis=0)\n",
    "        return y.T \n",
    "\n",
    "    x = x - np.max(x) # 溢出对策\n",
    "    return np.exp(x) / np.sum(np.exp(x))\n",
    "\n",
    "def cross_entropy_error(y, label):\n",
    "    if y.ndim == 1:\n",
    "        y = y.reshape(1, y.size)\n",
    "        label = label.reshape(1, label.size)\n",
    "\n",
    "    # 对输入的y进行处理，取batch\n",
    "    batch_size = y.shape[0]\n",
    "\n",
    "    # 交叉熵损失计算的是label（one-hot）中为1的那一位对应的输出，所以找出为1的索引\n",
    "    one_index = label.argmax(axis=1)\n",
    "\n",
    "    return -np.sum(np.log(np.add(y[np.arange(batch_size), one_index], 1e-7)))/batch_size\n",
    "    # 注意:y[np.arange(batch_size), one_index]不能写成y[:, one_index]\n",
    "\n",
    "class SoftmaxWithLoss:\n",
    "    def __init__(self) -> None:\n",
    "        self.loss = None\n",
    "        self.y = None\n",
    "        self.label = None\n",
    "\n",
    "    def forward(self, x, label):  # 此处x是一个向量\n",
    "        self.label = label\n",
    "        self.y = softmax(x)\n",
    "        self.loss = cross_entropy_error(self.y, self.label)\n",
    "\n",
    "        return self.loss\n",
    "\n",
    "    def backward(self, dout=1):\n",
    "        batch_size = self.label.shape[0]\n",
    "        if self.label.size == self.y.size: # 监督数据是one-hot-vector的情况\n",
    "            dx = (self.y - self.label) / batch_size\n",
    "        else:\n",
    "            dx = self.y.copy()\n",
    "            dx[np.arange(batch_size), self.label] -= 1\n",
    "            dx = dx / batch_size\n",
    "        \n",
    "        return dx\n",
    "        "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上述代码中有一点要提一下：**为什么在backward中要除以batch_size？**\n",
    "\n",
    "答：因为在当$batch\\_size=N(N>1)$时，上述的$Softmax\\_with\\_Loss$的计算图就要改一改了，如下：\n",
    "\n",
    "![image-20230326214925062](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230326214925062.png)\n",
    "\n",
    "最后会有一个除以$batch\\_size$的计算，所以对“/”节点进行反向传播，就会多一个$\\frac{1}{B\\_S}$的常数，该常数会一直传递到最前端，导致所有的反向传播输出都会变成$\\frac{1}{batch\\_size}(y_k-t_k)$。\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.5 误差反向传播的实现\n",
    "\n",
    "上面已经实现了Affine层、各种激活函数层，现在就可以把他们组装成一个神经网络了。\n",
    "\n",
    "现在再来回顾一下神经网络的步骤：\n",
    "\n",
    "1. 选取数据\n",
    "\n",
    "2. 计算梯度\n",
    "\n",
    "3. 更新参数\n",
    "\n",
    "4. 重复上述步骤\n",
    "\n",
    "![image-20230327090456066](https://zyc-learning-1309954661.cos.ap-nanjing.myqcloud.com/machine-learning-pic/image-20230327090456066.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "class TwoLayerNet:\n",
    "    def __init__(self, input_size, hidden_size, output_size, weight_init_std=0.01) -> None:\n",
    "\n",
    "        # 参数初始化\n",
    "        self.params = {}\n",
    "        self.params['W1'] = weight_init_std * \\\n",
    "            np.random.randn(input_size, hidden_size)\n",
    "        self.params['W2'] = weight_init_std * \\\n",
    "            np.random.randn(hidden_size, output_size)\n",
    "        self.params['b1'] = np.zeros(hidden_size)\n",
    "        self.params['b2'] = np.zeros(output_size)\n",
    "\n",
    "        # 搭建网络结构\n",
    "        self.layer = {}  # pyhton3.6之后dict有序，无需用OrderedDict\n",
    "        self.layer['Affine1'] = Affine(self.params['W1'], self.params['b1'])\n",
    "        self.layer['Relu1'] = Relu()\n",
    "        self.layer['Affine2'] = Affine(self.params['W2'], self.params['b2'])\n",
    "        # 为什么要softmaxwithloss要单独用一个属性lastlayer？因为它的forward函数参数和前面的不一样，除了x还有label，之后不能统一用for循环\n",
    "        self.lastLayer = SoftmaxWithLoss()\n",
    "\n",
    "    def predict(self, input):\n",
    "        for layer in self.layer.values():\n",
    "            input = layer.forward(input)\n",
    "\n",
    "        return input\n",
    "\n",
    "    def loss(self, input, label):\n",
    "        output = self.predict(input)\n",
    "        loss = self.lastLayer.forward(output, label)\n",
    "\n",
    "        return loss\n",
    "\n",
    "    def accuracy(self, input, label):\n",
    "        y = self.predict(input)\n",
    "        if y.ndim == 1:\n",
    "            y.reshape(1, y.size)\n",
    "            label.reshape(1, label.size)\n",
    "\n",
    "        batch_size = y.shape[0]\n",
    "        y_index = np.argmax(y, axis=1)\n",
    "        label_index = np.argmax(label, axis=1)\n",
    "\n",
    "        return np.sum(y_index == label_index)/float(batch_size)\n",
    "\n",
    "    def gradient(self, input, label):\n",
    "        # 正向传播\n",
    "        loss=self.loss(input,label)\n",
    "\n",
    "        # 反向传播\n",
    "        dout = 1\n",
    "        dout = self.lastLayer.backward(dout)\n",
    "        layers = list(self.layer.values())\n",
    "        layers.reverse()\n",
    "\n",
    "        for layer in layers:\n",
    "            dout = layer.backward(dout)\n",
    "        # 保存梯度\n",
    "        grad = {}\n",
    "        grad['W1'] = self.layer['Affine1'].dW\n",
    "        grad['b1'] = self.layer['Affine1'].db\n",
    "        grad['W2'] = self.layer['Affine2'].dW\n",
    "        grad['b2'] = self.layer['Affine2'].db\n",
    "\n",
    "        return grad,loss\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.6利用TwoLayerNet学习"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train_acc: 0.08466666666666667 \t test_acc: 0.0831 \t loss: 2.302277416545889\n",
      "train_acc: 0.9030666666666667 \t test_acc: 0.905 \t loss: 0.30658978808740867\n",
      "train_acc: 0.9213833333333333 \t test_acc: 0.9242 \t loss: 0.20852397688553626\n",
      "train_acc: 0.9341333333333334 \t test_acc: 0.9326 \t loss: 0.18906242815935798\n",
      "train_acc: 0.9457333333333333 \t test_acc: 0.9426 \t loss: 0.3013873673649449\n",
      "train_acc: 0.9528 \t test_acc: 0.9506 \t loss: 0.15033393637781933\n",
      "train_acc: 0.9589166666666666 \t test_acc: 0.9544 \t loss: 0.045285282803728906\n",
      "train_acc: 0.9626333333333333 \t test_acc: 0.9585 \t loss: 0.12097963389795147\n",
      "train_acc: 0.9639666666666666 \t test_acc: 0.9589 \t loss: 0.10043134002144184\n",
      "train_acc: 0.9677166666666667 \t test_acc: 0.9614 \t loss: 0.13287334005675022\n",
      "train_acc: 0.9700333333333333 \t test_acc: 0.9639 \t loss: 0.05351279126316677\n",
      "train_acc: 0.9714666666666667 \t test_acc: 0.9653 \t loss: 0.1347767476149933\n",
      "train_acc: 0.9728166666666667 \t test_acc: 0.9653 \t loss: 0.06633202947311678\n",
      "train_acc: 0.9756333333333334 \t test_acc: 0.9682 \t loss: 0.02790441184341367\n",
      "train_acc: 0.9757833333333333 \t test_acc: 0.9664 \t loss: 0.11658530955890015\n",
      "train_acc: 0.9777 \t test_acc: 0.9679 \t loss: 0.1852543469083793\n",
      "train_acc: 0.9782 \t test_acc: 0.9672 \t loss: 0.143177571583826\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 640x480 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import sys,os\n",
    "\n",
    "current_file_path = os.path.dirname(os.path.realpath('__file__'))\n",
    "Demo_path = os.path.join(current_file_path, 'Demo')\n",
    "sys.path.append(Demo_path)\n",
    "\n",
    "import numpy as np\n",
    "from dataset.mnist import load_mnist\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# 读取数据\n",
    "(x_train, label_train), (x_test, label_test) = load_mnist(\n",
    "    normalize=True, flatten=True, one_hot_label=True)\n",
    "\n",
    "# 设置超参数\n",
    "learning_rate = 0.1\n",
    "batch_size = 100\n",
    "train_size = x_train.shape[0]\n",
    "iter_num = 10000\n",
    "iter_per_epoch = max(train_size/batch_size, 1)\n",
    "\n",
    "loss_list = []\n",
    "train_accuracy_list = []\n",
    "test_accuracy_list = []\n",
    "\n",
    "# 神经网络学习\n",
    "network = TwoLayerNet(\n",
    "    input_size=x_train.shape[1], hidden_size=50, output_size=label_train.shape[1])\n",
    "\n",
    "for i in range(iter_num):\n",
    "    # 划分数据集\n",
    "    batch_mask = np.random.choice(train_size, batch_size)\n",
    "    x_batch = x_train[batch_mask]\n",
    "    label_batch = label_train[batch_mask]\n",
    "    # 求梯度和损失值\n",
    "    grad, loss = network.gradient(x_batch, label_batch)\n",
    "    # 更新参数\n",
    "    for key in (grad.keys()):\n",
    "        network.params[key] -= learning_rate*grad[key]\n",
    "\n",
    "    # 计算loss和accuracy\n",
    "    if i % iter_per_epoch == 0:\n",
    "        train_acc = network.accuracy(x_train, label_train)\n",
    "        test_acc = network.accuracy(x_test, label_test)\n",
    "        loss_list.append(loss)\n",
    "        train_accuracy_list.append(train_acc)\n",
    "        test_accuracy_list.append(test_acc)\n",
    "        print('train_acc:',train_acc,'\\t test_acc:',test_acc,'\\t loss:',loss)\n",
    "\n",
    "# 画图保存在img文件夹里\n",
    "plt.plot(np.arange(len(loss_list)), loss_list)\n",
    "plt.title('loss curve')\n",
    "plt.xlabel('epoch')\n",
    "plt.ylabel('loss')\n",
    "plt.savefig('img/ch4_loss.png')\n",
    "plt.clf()\n",
    "\n",
    "plt.plot(np.arange(len(train_accuracy_list)), train_accuracy_list, label='train_acc')\n",
    "plt.plot(np.arange(len(test_accuracy_list)), test_accuracy_list, label='test_acc')\n",
    "plt.title('train/test_acc curve')\n",
    "plt.xlabel('epoch')\n",
    "plt.ylabel('train/test_acc')\n",
    "plt.legend()\n",
    "plt.savefig('img/ch4_train_test_acc.png')\n",
    "plt.clf()\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pythonProject",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
